System for multi-sensor image fusion

ABSTRACT

A method and system for fusing image data includes an electronic circuit (L) for synchronizing image frames. An adaptive lookup table unit ( 302   a,    302   b ) receives the image frame data sets and applies correction factors to individual pixels. The size of an image is then scaled or otherwise manipulated as desired by data formatting processors ( 312   a  and  312   b ). Multiple images communicated within parallel circuit branches (P 1  and P 2 ) are aligned and registered together with sub-pixel resolution.

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. ProvisionalApplication Serial No. 60/319,307, filed Jun. 12, 2002, entitled SYSTEMFOR MULTISENSOR IMAGE FUSION.

BACKGROUND OF INVENTION

[0002] 1. Technical Field

[0003] This invention relates generally to the field of imaging systemsand more specifically to a method and system for fusing image data frommultiple sources.

[0004] 2. Background Art

[0005] Multiple sensor imaging systems generate an image of an object byfusing data that is collected using multiple sensors. Gathering imagedata using multiple sensors, however, has posed challenges. In somesystems, the sensors detect light received from separate apertures. Datagenerated from light from separate apertures, however, describedifferent points of view of an object that need to be reconciled inorder to fuse the data into a single image. Additionally, using separateapertures for different sensors may increase the bulk of an imagingsystem.

[0006] In other systems, light from an aperture is split into componentsbefore entering the sensors. Reflective and refractive elements aretypically used to direct the light to different sensors. For example,the system described in U.S. Pat. No. 5,729,376 to Hall et al. includesmultiple reflective and refractive elements such as a lens that reflectslight towards one sensor and refracts light towards another sensor. Eachindividual sensor, however, detects only a component of light, forexample, only specific wavelengths of light, and thus cannot generateimage data from the full spectrum. Additionally, multiple reflective andrefractive elements may add to the bulk and weight of an imaging system.Consequently, gathering image data from multiple sensors has posedchallenges for the design of imaging systems.

[0007] While the above cited references introduce and disclose a numberof noteworthy advances and technological improvements within the art,none completely fulfills the specific objectives achieved by thisinvention.

SUMMARY OF INVENTION

[0008] While known approaches have provided improvements over priorapproaches, the challenges in the field of imaging systems havecontinued to increase with demands for more and better techniques havinggreater effectiveness. Therefore, a need has arisen for new methods andsystems for gathering image data using multiple sensors.

[0009] In accordance with the present invention, a method and system foradaptively fusing a plurality of images, such as video or one or morestill images, from a plurality of sources are provided thatsubstantially eliminate or reduce the disadvantages and problemsassociated with previously disclosed systems and methods.

[0010] The method and system for fusing image data includessynchronizing image frames. An adaptive lookup table is used to applycorrection factors to individual pixels in an image. The size of animage is then scaled. Multiple images are aligned and registeredtogether with sub-pixel resolution.

[0011] These and other objects, advantages and features of thisinvention will be apparent from the following description taken withreference to the accompanying drawings, wherein is shown the preferredembodiments of the invention.

BRIEF DESCRIPTION OF DRAWINGS

[0012] A more particular description of the invention briefly summarizedabove is available from the exemplary embodiments illustrated in thedrawings and discussed in further detail below. Through this reference,it can be seen how the above cited features, as well as others that willbecome apparent, are obtained and can be understood in detail. Thedrawings nevertheless illustrate only typical, preferred embodiments ofthe invention and are not to be considered limiting of its scope as theinvention may admit to other equally effective embodiments.

[0013]FIGS. 1A and 1B illustrate one embodiment of a system forgathering image data.

[0014]FIG. 2 illustrates one embodiment of a system for gathering imagedata that includes three or more sensors.

[0015]FIG. 3 is a flowchart demonstrating one embodiment of a methodthat may be used with the system of FIG. 1.

[0016]FIG. 4 is a flowchart demonstrating one embodiment of the presentmethod for fusing a plurality of images.

DETAILED DESCRIPTION

[0017] So that the manner in which the above recited features,advantages, and objects of the present invention are attained can beunderstood in detail, more particular description of the invention,briefly summarized above, may be had by reference to the embodimentthereof that is illustrated in the appended drawings. In all thedrawings, identical numbers represent the same elements.

[0018]FIG. 1A illustrates a side view of one embodiment of a system 100for gathering image data. System 100 receives light reflected from anobject 110 and gathers information from the light to generate an imageof object 110 on a display 142. System 100 may include an outer casing112 having an aperture 114 through which light enters. Outer casing 112may have any suitable shape such as a cylinder having a diameter in therange of 8-12 cm, for example, approximately 10 cm, and a length in therange of 12-15 cm, for example, approximately 14 cm. System 100 may alsoinclude an inner assembly 116 coupled to outer casing 112 with braces124 as illustrated in FIG. 1B. FIG. 1B illustrates a front view of innerassembly 116 coupled to casing 112 with braces 124.

[0019] Referring to FIG. 1A, inner assembly 116 may include optics 118and a sensor 120, each of which may be coupled to an inner casing 117.Inner casing 117 may have any suitable shape such as a cylinder having adiameter in the range of 3 to 6 cm, for example, approximately 4.5 cm,and a length in the range of 7 to 10 cm, for example, approximately 8 cmin length. Optics 118 focuses light reflected from object 110 ontosensor 120. Optics 118 may include, for example, a lens comprising glassor polymer having a radius in the range of 3 to 5 cm, for example,approximately 4 cm, and a focal length in the range of 20-22 mm, forexample, approximately 22 mm. Optics 118, however, may include anysuitable optical element or configuration of optical elements forfocusing light from object 110 onto sensor 120.

[0020] Sensor 120 detects the light reflected from object 110 directlythrough aperture 114, that is, through an uninterrupted pathway. Sensor120 may be placed such that sensor 120 receives light generally in adirection that light travels from object 110 to aperture 114. Sensor 120may detect certain types of energy, for example, infrared energy, of thelight. Sensor 120 may enhance certain features of light such as, forexample, an image intensifier sensor. Sensor 120, however, may compriseany suitable sensor, for example, a long wave infrared sensor, a lowlight level charge coupled device (LLLCCD), or a complementarymetal-oxide semiconductor (CMOS) sensor.

[0021] Sensor 120 generates sensor data set S1 in response to thereceived light. Sensor data set S1 may include values assigned to pixelscorresponding to points of light, where the values represent imageinformation such as brightness or color associated with the points oflight. Sensor 120 transmits sensor data set S1 to a fusing module 140.

[0022] System 100 may also include an outer assembly 138 comprisingreflective surfaces 130 and 132 and a sensor 134. Reflective surface 130and sensor 134 may be coupled to outer casing 112, and reflectivesurface 132 may be coupled to inner casing 117. Any suitableconfiguration, however, may be used, for example, outer assembly 138 maybe configured as a Schmidt-Cassegran catadioptric optical assembly, adiffractive optical system, or any combination of suitableconfigurations.

[0023] Reflective surface 130 receives light from object 110 throughaperture 114 and reflects the received light. Reflective surface 130 maycomprise a metallic or dichroic mirror having a diameter in the range of8 to 10 cm, for example, approximately 9 cm and a focal length in therange of 24 to 26 mm, for example, approximately 25 mm. Reflectivesurface 130, however, may comprise any material and may have any shapesuitable for receiving light through aperture 114 and reflecting lightto reflective surface 132. Reflective surface 132 receives light fromreflective surface 130 and reflects the received light. Reflectivesurface 132 may comprise a metallic or dichroic mirror having a diameterin the range of 7 to 10 cm, for example, approximately 8 cm and a focallength in the range of 24 to 26 cm, for example, approximately 25 mm.Reflective surface 132, however, may comprise any material and may haveany shape suitable for receiving light from reflective surface 130 andreflecting light to a receptor area 133 of sensor 134.

[0024] Receptor area 133 of sensor 134 detects light reflected fromreflective surface 132. Sensor 134 may include, for example, an infraredsensor or an image intensifier sensor. Sensor 134, however, may compriseany suitable sensor, for example, a long wave infrared sensor, a mediumwave infrared sensor, a short wave infrared sensor, a low light levelcharge coupled device (LLLCCD), or a complementary metal-oxidesemiconductor (CMOS) sensor. Sensor 134 generates sensor data set S2 inresponse to the received light. Sensor 134 may generate a different typeof data set than that generated by sensor 120. For example, sensor 120may include an infrared sensor that detects infrared energy of receivedlight to generate a data set, and sensor 134 may include an imageintensifier sensor that enhances certain features of received light togenerate a different type of data set. Sensor data set S2 may includevalues assigned to pixels corresponding to points of light, where thevalues represent image information associated with the points of light.Sensor 134 transmits sensor data S2 to fusing module 140.

[0025] System 100 may have a central axis 136 located approximatelyalong a light path from object 110 to receptor area 133 of sensor 134.Sensor 120 and sensor 134 may be substantially coaxial such that sensor120 and sensor 134 receive light at a point approximately along centralaxis 136. Sensor 120 and sensor 134 may be configured such that thediameter of inner assembly 116 is less than the diameter of reflectivesurface 130, and inner assembly 116 is approximately centered overreflective surface 130 as illustrated in FIG. 1B. FIG. 1C illustrates afront view of system 100 where inner assembly 116 is approximatelycentered in front of reflective surface 130. In the illustratedembodiment, the configuration of sensors 120 and 134 allows sensors 120and 134 to receive light from the same aperture with minimal reflectiveand refractive elements, providing for a compact imaging system.

[0026] Fusing module 140 receives sensor data S1 and S2 from sensors 120and 134, respectively. Fusing module 140 fuses sensor data sets S1 andS2 to generate fused data. For example, fusing module 140 combinesvalues of sensor data sets S1 and S2 for pixels corresponding to thesame point of light to generate the fused data. Fusing module 140 mayuse any suitable process for fusing data sets S1 and S2, for example,digital imaging processing, optical overlay, or analog video processing.

[0027] In the illustrated embodiment, sensor 120 and sensor 134 detectlight received through the same aperture 114, so both sensors 120 and134 receive light describing the same point of view of object 110. As aresult, fusing module 140 does not need to perform data processing toreconcile different points of view. Additionally, since minimalreflective and refractive elements are used, the light detected bysensors 120 and 134 undergoes few changes. As a result, fusing module140 does not need to perform processing to compensate for changes due tomultiple reflective and refractive elements.

[0028] Display 142 receives the fused data from fusing module 140, andgenerates an image of object 110 using the fused data. Display 142 mayinclude any suitable system for displaying image data, such as anorganic light-emitting diode (OLED), nematic liquid-crystal display(LCD), or field emitting display (FED), in panel display, eyepiecedisplay, or near-to-eye display formats.

[0029] Although the illustrated embodiment shows two sensors 120 and134, the system of the present invention may include any suitable numberof sensors, as described in connection with FIG. 2.

[0030]FIG. 2 is a block diagram of one embodiment of a system 200 thatincludes three sensors for gathering image data. System 200 includes aninner assembly 216 coupled to an outer casing 212. Inner assembly may besubstantially similar to system 100 of FIG. 1, which includes twosensors 120 and 134. Outer assembly 238 may be substantially similar toouter assembly 138. That is, reflective surfaces 230 and 232, which maybe substantially similar to reflective surfaces 130 and 132,respectively, are coupled to inner assembly 216 and outer casing 212,respectively. Additionally, sensor 234, which may be substantiallysimilar to sensor 134, is coupled to outer casing 212. Sensors 120, 134,and 234 may be substantially coaxial. Fusing module 140 is coupled tosensors 120, 134, and 234, and display 142 is coupled to fusing module140.

[0031] In operation, system 200 receives light reflected from object110. Inner assembly 216 may generate data sets S1 and S2 in a mannersubstantially similar to that of system 100 of FIG. 1. Sensor 234receives light reflected from reflective surfaces 230 and 232 in asubstantially similar matter to that of sensor 134 to generate datasetS3. Fusing module 140 receives datasets S1, S2 and S3 and fuses thedatasets to generate fused data. Display 142 receives the fused data andgenerates an image from the fused data. Additional sensors may be addedto system 200.

[0032]FIG. 3 is a flowchart illustrating one embodiment of a method forgathering image data using system 100 of FIG. 1. The method begins atstep 210, where light reflected from object 110 is received by aperture114. The reflected light includes image information that may be used toform an image of object 110. At step 212, sensor 120 detects thereceived light. Optics 118 may be used to focus the light onto sensor120. Sensor 120 generates a data set S1 from the detected light andtransmits data set S1 to fusing module 140 at step 214. Sensor 120 may,for example, detect infrared light reflected from object 110 andgenerate a data set S1 that describes the infrared light.

[0033] At step 216, reflective surface 130 receives light from object110 and reflects the received light to reflective surface 132.Reflective surface 132 receives the reflected light and, in turn,reflects the received light to sensor 134 at step 218. At step 220,sensor 134 detects light reflected from reflective surface 132. Sensor134 generates data set S2 from the received light at step 222. Sensor134 may include an image intensifier sensor that enhances certainfeatures of the light received from object 110, and may generate a dataset that describes the enhanced features.

[0034] At step 224, fusing module 140 receives data sets S1 and S2 andfuses the received data sets to generate fused data. Fusing module 140may, for example, combine values from data sets S1 and S2 for pixelscorresponding to the same point of light. Display 142 receives the fuseddata and then displays an image of object 110 at step 226. Afterdisplaying the image, the method terminates.

[0035] The present method and system for fusing image data shown in FIG.4 includes an electrical circuit L for synchronizing image framesrepresentative of data collections from at least two data streams orsets S1 and S2. An adaptive lookup table component 302 a and 302 b isused to apply correction factors to individual pixels or data records orpoints in an image. The size of an image or data collection grouping isthen scaled or formatted by the data formatting component or step 312 aor 312 b. Multiple images generally are aligned and registered togetherwith sub-pixel resolution in the data fusion unit 304 of the circuit L.

[0036] Image Frame Synchronization.

[0037] A method for synchronizing multiple images from multiple sourcestogether so that they can be later fused into a single image is furtherdisclosed herein. As the multiple images are received or retrieved,selected image data records are separately clipped, extended, and scaledas needed to a common output resolution (or integer multiple of) orother common characteristic appropriate for producing the desiredresulting output data stream S3 to be viewed or further processed.

[0038] The initially processed images are then communicated to andstored in separate buffers or electro-optical memory units. Twocomparable buffer units or process steps 306 a and 306 b are shownbuffering data sets S1 and S2, respectively, in parallel circuitbranches P1 and P2 of circuit L. Two circuit branches P1 and P2 areshown by way of example in FIG. 4, although a plurality of branches maybe designed into circuit L to accommodate a desired number of datarecords to be fused or processed together to form the output set S3.

[0039] Each buffer may be large enough or have the capacity to hold anentire image frame or hold partial images that are misaligned in time bya fixed delay. The processed images or selected data records are readfrom the buffers 306 a and 306 b in parallel and are eventually combinedby a fusion multiplexer 304 as the data is sent to a storage device or adisplay device depicted schematically as an output set or signal S3 inFIG. 4.

[0040] Adaptive Lookup Table.

[0041] An adaptive lookup table 302 a and 302 b may be implemented toprovide a fast, hardware efficient method for applying correctionfactors to individual pixels or selected data records in an imagerecord. A correction factor stored within the correction circuitcomponent or step 302 a or 302 b may be applied, which correction factorcan include gamma correction, brightness adjustment, contrastadjustment, pixel bit-width conversion, an image fusion ratio, or thelike.

[0042] As each pixel or data portion of an image is received, thepixel's value is generally electro-optically communicated or input tothe lookup table as an address to table units 302 a or 302 b. Thecontents at the sent address are transmitted or used as one or moreoutput datum points, and are used in the output image S3 in place of theinput pixel values.

[0043] The lookup table 302 a or 302 b is desirably adaptively updatedby an algorithm which is based on metrics of the current image datagrouping, previous image metrics, other image source data, userpreferences, system settings, and metrics from other image sources. Suchmetric collection is shown schematically in FIG. 4 as metric collectioncomponents 308 a and 308 b of circuit branches P1 and P2, respectively.The aforementioned algorithm may be implemented in hardware or insoftware.

[0044] A pixel bit-width conversion may also be accomplished by makingthe input bit width different from the output bit width. The output bitwidth may be made either larger or smaller than the input bit width, asdesired for the specific system's needs for which the logic circuit Lforms a part.

[0045] Image Scaling.

[0046] An optional method of scaling the size of an image at video framerates is further disclosed that is computationally efficient andrequires minimum resources to implement.

[0047] The present scaling method also provides a convenient point todecouple an incoming image pixel clock from a system pixel clock. Thisallows the scaling method to operate at a faster rate than the rate ofthe incoming pixel data stream. The primary advantage of the presentmethod is to allow the scaled output resolution to be greater than theincoming resolution when operating in an interpolation mode.

[0048] Specifically, the first two or more scan lines of the incomingimage preferably are each stored in a scan line memory buffer, such ascollection components 308 a and 308 b or other electro-optical circuitelement as chosen by the designer. Two lookup tables are preferablyutilized to hold computation coefficients and pixel pointer indexes: ahorizontal table, and a vertical table.

[0049] The table values may be pre-computed and loaded when requested.Alternatively, the table values could be computed in the background byan attached microprocessor 310 or other processing component. The tablevalues may contain electro-optically stored information such as thestarting pixel index, multiplication coefficients, and pixel indexincrement values. The scaling method may be implemented as a statemachine that steps through the tables scan-line by scan-line. Thecoefficients in the tables are used to perform either a linear ordecimation of the pixels pointed to by the table such as by format dataprocessors 312 a or 312 b, and produce a new pixel value at the desiredoutput resolution for output stream S3.

[0050] The present scaling method is further capable of scaling an imageto a higher resolution or to a lower resolution than the inputresolution. The scaling method herein is designed to provide sub-pixelscaling such that non-integer scaling factors can be used. Theresolution is generally limited by the number of bits available in theelectro-optical lookup tables and the number of bits used in themultiplication stages of the scalar component.

[0051] Image Alignment and Registration.

[0052] A present method for aligning and registering multiple imagerecords together with sub-pixel resolution generally includes using theaforementioned scaling method to stretch or shift each of the outputimage data records in horizontal and vertical directions. The scalingfactors and image shift offsets may be determined by a manual usercalibration sequence or by an automatic calibration sequence.

[0053] An electronic imager (CMOS, CCD, etc) may be used toelectronically sample an image record that is displayed on or by animage intensifier, which may form a part of the sensor units describewith regard to FIGS. 1A and 2. A horizontal and a vertical deflectorcoil may be placed on the intensifier tube. A pair of AC controlsignals, which signals are synchronized to an electronic imager framerate, drive the deflector coils, so as to slightly move the image on theintensifier tube up/down or left/right by an amount equal to one-half,one-quarter, etc. of the size of a pixel in the electronic imager.

[0054] After the new position of the image on the tube stabilizes, theelectronic imager would capture a new field. Several fields may besampled with the image moved each time by a partial scan line height anda partial pixel width. The resulting captured fields can either bedisplayed as an interlaced output signal or combined into a singlelarger image that contains 2×, 4×, etc. the number of actual pixels inthe sensor.

[0055] A second alternative embodiment uses deflector coils on a tube,which deflector coils could be used to affect an image alignmentmechanism. An adjustable signal may then be driven into the deflectioncoils that would deflect the displayed or output image by a constantamount.

[0056] The foregoing disclosure and description of the invention areillustrative and explanatory thereof, and various changes in the size,shape and materials, as well as in the details of the illustratedconstruction may be made without departing from the spirit of theinvention.

1. A method for fusing image data from two or more data sets into anoutput data stream, the method including the steps of: synchronizingimage frame data records; using an adaptive lookup table to applycorrection factors to individual pixels in an image to be processed; andaligning and registering multiple selected synchronized images together.2. The method of claim [claim Reference] wherein a fusion decisionprocessor aligns and registers the multiple synchronized images.
 3. Themethod of claim 1 further including scaling the size of an image recordthat has been processed with the adaptive lookup table.
 4. The method ofclaim [claim Reference] wherein individual data streams are formattedfollowing look up table processing and before the multiple data streamsare fused.
 5. The method of claim [claim Reference] wherein thecorrection factors applied are selected from the group consisting ofgamma correction, brightness adjustment, contrast adjustment, pixelbit-width conversion or image fusion ratio.
 6. An image data set fusionsystem for producing an output data stream by fusing image data recordsfrom two or more data sets comprising:_Ref42655739 a plurality ofparallel circuit branches each of which branches receive a data streamto be processed; a data fusion processor component for controllablyfusing image data records communicated from the parallel circuitbranches; and at least one of the parallel circuit branches having alook up table correction processing module to receive an input datastream corresponding to the parallel circuit branch and to apply desiredcorrection parameters to the input data stream.
 7. The invention ofclaim [claim Reference] wherein the fusion decision processor aligns andregisters the multiple synchronized images.
 8. The invention of claim[claim Reference] further including scaling the size of an image recordthat has been processed with the adaptive lookup table.
 9. The inventionof claim [claim Reference] wherein the correction factors applied by thelook up table correction processing module are selected from the groupconsisting of gamma correction, brightness adjustment, contrastadjustment, pixel bit-width conversion or image fusion ratio.